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One of the most basic and obvious sources of data for education policy analysis is 
text. This article discusses content analysis as an important part of the 
methodological toolbox for elucidating patterns and trends about education 
policy. Focusing specifically on media, I show hozv media content analysis can 
produce nuanced insights about the ways in which educational ideas are 
understood in a local context. Drawing on a research project that analyzed 
thirteen years of media coverage of the Programme for International Student 
Assessment (PISA), I demonstrate the utility of media content analysis for 
understanding the way in which ideas and global trends influence national 
education policy on the ground. 


Introduction 

Much comparative education scholarship of the past decade has concerned itself with 
understanding how the global and the local interact to produce education policy. These 
studies have used a variety of methods to try and understand how actors, institutions, 
ideas and interests combine to influence education politics and policy. Empirically, this 
work has taken a variety of forms, from large-n quantitative studies to the deep 
exploration of a single case. One method that has recently been employed to understand 
a particular phenomenon across different contexts is content analysis. This is not 
surprising in that text is perhaps the most obvious source material in the study of 
education policy. In this article, I focus on one type of content analysis, namely media 
content analysis, and propose it as a fruitful method for understanding the construction 
of educational issues in public life. Drawing on contemporary comparative education 
research for examples, I show that media content analysis can help comparativists 
understand how public communication influences and reflects understandings about 
education in the public sphere. 

The article proceeds in four parts. First, I define content analysis and discuss how it 
relates to other methods of textual analysis, namely discourse analysis. I then focus 
specifically on media analysis as a useful method for understanding policy context. I 
provide an overview of the role of media in the policy process generally and offer some 
recent examples of comparative education research that have used media content 
analysis to explore the impacts of international large-scale assessments. In the third 
section, I present an example of a media analysis of the Programme for International 
Student Assessment (PISA) in the United States, and share specific ideas about how a 
researcher might approach conducting a media content analysis of this nature, from 
theoretical framing through data analysis. The last section discusses limitations of media 
content analysis and concludes. 
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What is Content Analysis? 

Content analysis refers to a general set of techniques for analyzing collections of 
communications. In Lasswell's classic construction, content analysis examines "who says 
what through which channel to whom with what effect" (Lasswell, 1948, p. 117). 
Perhaps the most distinctive characteristic of content analysis as compared to other 
types of analysis is that its data derives from communicative practices. As such, it is 
most widely used to collect and analyze data to understand the meanings ascribed to an 
issue within a given context (Krippendorf, 1989, p. 403). This process involves 
discerning meaning about attitudes, symbols, cultures and institutions from which 
inferences are ultimately drawn. The analysis is often not of the literal description of the 
content, but rather the illumination of patterns and trends that are not immediately 
observable. Many different types of content may be used for analysis. Most content 
analysis is textual document analysis, which analyzes any type of printed materials, 
(e.g., newspapers, magazines, letters, books, testimony, governmental publications, 
statutes, etc.) but other sources (e.g., films, radio broadcasts, television programming, 
etc.) are also appropriate. 

There is some disagreement about the place of content analysis in the methodological 
toolbox. 1 Content analysis is often juxtaposed with discourse analysis in terms of 
ontology and epistemology; while both examine textual sources, content analysis is seen 
as trying to uncover reality as it exists, while discourse analysis is seen as trying to 
uncover reality as it is produced (Hardy, Harley and Phillips, 2004). In this opposition, 
content analysis is positivist, objective, and quantitative while discourse analysis is 
interpretivist, intersubjective and qualitative. Generally, content analysis proceeds from 
an understanding that meaning can be counted and coded, and content analyses use a 
codebook with an a priori coding scheme that allows the researcher to map the patterns 
and meaning of particular content from which inferences can be drawn (Lowe, 2004). As 
with all empirical research, content analysis methods rely on systematic and replicable 
techniques to generate data for investigation. Advocates of content analysis highlight 
that the strength of the method is in its reliability and replicability; if the analytic 
categories and coding scheme are properly designed, anyone should be able to conduct 
the analysis (Krippendorf, 1989; Neuendorf, 2004). This constrains the degree to which 
an individual researcher's views can mediate the interpretation of data. A more 
moderate ground can be found with those who suggest a slightly relaxed understanding 
of the relationship between content and discourse analysis, whereby prior empirical 
work and the scholar's research questions provide the categories for research but 
additional valid analytical constructs can emerge as the data is collected and analyzed 
(Hardy, Harley and Phillips, 2004). 

A prominent form of content analysis is media analysis because media are generally 
acknowledged to play a key role in interpreting and disseminating ideas about public 
policy. Media content analysis can be an economical form of data collection, since much 
media data is available online or, for the academic researcher, through subscription 


1 See Macnamara (2005) for a discussion of the tensions about whether media analysis is best 
suited to quantitative or qualitative use. 
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search services. Given the ubiquity of electronic data and archives, it is fairly easy to 
locate and collect primary data. Though social media are undoubtedly profoundly 
changing the way education policy communication happens (see Supowitz, Daly, & Del 
Fresno, 2015), this article focuses on the analysis of traditional print media (newspapers), 
in both their print and electronic forms, as these media have been the source of most 
comparative education media analysis to date. 

Media Content Analysis 

Media are recognized in the policy studies literature as playing an important, perhaps 
key, role in the policy process, as both purveyors of information and as ciphers for 
competing ideas. Media are usually accorded a prime role in the process of policy 
agenda setting. Hallin and Mancini (2004) caution that most research about the media 
and the policy process treats media as a monolith, one that operates similarly in all 
contexts, when in fact the media are rooted in specific political and economic contexts 
and behave according to those local logics. 

Research has demonstrated the key role that media play in political agenda setting by 
choosing which stories and issues are reported on, and then how those issues are 
covered (McCombs & Shaw, 1974; Gamson & Modigliani, 1989; Gamson, 1992). Media 
coverage is theorized to both reflect and create public policy and public opinion. While 
research suggests varying degrees to which media actually influence the policy process, 2 
it is clear that media do affect how issues come to be understood as public issues, 
reflecting broader cultural, historical and institutional affinities. 

Media operate in at least two ways to define public problems. The first is through 
framing. Framing refers to the ways in which issues are organized and understood in 
the public arena; that is, frames are the organizing ideas, words, images and themes that 
are used to describe and structure information about a public policy issue. Issue 
framing is a key element of political discourse and policymaking, and has been shown to 
have an impact on attitudes and policy preferences amongst voters, politicians and 
journalists (Chong & Druckman, 2007). The concept of framing is analytically useful in 
illuminating how ideas are generated, diffused and mobilized (Benford & Snow, 2000). 

The second way media influence the policy process is in playing a gatekeeping role in 
whom is given status to comment on public problems and prescribe solutions. The 
conferral of status is captured in the concept of standing. As Ferree, Gamson, Gerhards 
& Rucht (2002) note: 

"[Standing] refers to gaining the status of a regular media source whose 
interpretations are directly quoted. Standing is not identical to receiving 
any sort of coverage or mention in the news; a group may appear when it 
is described or criticized but still have no opportunity to provide its own 
interpretation and meaning to the event in which it is involved. Standing 


2 For instance, as Flenig points out, two of the most widely read books on the politics of agenda 
setting (Kingdon, 1995 and Baumgartner and Jones, 1993) disagree about the degree of the 
media's influence (Henig, 2008, p. 180, footnote 5). 
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refers to a group being treated as an actor with voice, not merely as an 
object being discussed by others." (p. 13) 

An analysis of standing views actors as signifying agents who are actively engaged in 
constructing meaning about social ideas. Standing, sometimes also referred to as 'voice/ 
is an essential component of policy discourse since it determines which actors have 
legitimacy and power. Standing can also reflect media convention, or editorial priorities, 
as well as media savvy. Not every actor has an equal chance of gaining standing; some 
actors are "better prepared and motivated to speak out on a particular topic, but the 
customary practices of news gathering make some speakers highly salient to the media 
while others are less so." (Ferree, et.al, 2002, p. 86) 

Media content analysis in comparative education research 

Recently comparative education scholars have turned to media analysis as a route to 
understanding the ways in which historical, institutional, cultural and political contexts 
combine to influence education policy. For instance, as international assessment has 
become a topic of scholarship in comparative education, several researchers have turned 
to media analysis to expand their understanding of how, why and under what 
conditions international assessments are used across varying national contexts. Much of 
this work has focused on one specific assessment, the Programme for International 
Student Assessment (PISA). PISA is an international assessment administered every 
three years by the Organisation for Economic Cooperation and Development (OECD) to 
a representative sample of 15-year old students from participating countries. The 
assessment measures student performance in three subject areas, mathematics, and 
reading and science literacy. 

For instance, Takayama (2010) uses media articles and other textual sources to examine 
how Japanese politicians and education actors use the media to construct an education 
"crisis" in Japan. Takayama, Waldow and Sung (2013; 2014) draw on media accounts in 
three countries to provide a comparative analysis of the responses to PISA results in 
Australia, Germany and South Korea. A team of European researchers used media 
accounts, along with other sources, to compare the effects of international assessments 
across six different country contexts (www.knowandpol.eu), and other recent work (e.g.. 
Martens and Niemann (2010); Fladmoe (2011); Dixon, Arndt, Mullers, Vakkuri, Englom- 
Pelkkala, and Hood (2013); Dobbins and Martens (2012) also analyzes press responses to 
rankings on PISA. The research findings are mixed, but each accentuates the role of the 
press in PISA reception and the analyses generally show the importance of national 
education politics and culture in shaping the press coverage of PISA. 


What Can We Learn From Media Content Analysis? PISA and U.S. Education 
Discourse 

In order to illustrate how a researcher might conduct a media content analysis and what 
such an analysis might expose, this section uses specific examples from my own work 
on a media content analysis of American media regarding the Programme for 
International Student Assessment (PISA) (Green Saraisky, 2015). This project sought to 
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understand the process of reception; that is, how PISA, as an internationally developed 
and administered assessment instrument, is understood in a local (in this case, national) 
context. The research grows out of the aforementioned academic literature that suggests 
that rankings and performance indicators can shape national education discourse in 
important ways. I was interested in evaluating how PISA is used in education policy 
debate, and in whether or how PISA has influenced American education policy. 

The media analysis must first be located within a theoretical framework that situates the 
research questions and also provides a rationale for why media content analysis is an 
appropriate method for answering those questions. In this case, the research aimed to 
understand the ways in which PISA results were being used in American education 
discourse: who uses PISA, and to what end? When are references to PISA being 
activated? How are ideas about educational success constructed in the public arena? 
These research questions were based in theory from political science and political 
sociology, as well as an interpretive framework about reception from comparative 
education (Steiner-Khamsi, 2014). They provided the scaffolding for the analysis and 
guided the development of the coding protocol, which incorporated variables that 
captured both framing and standing elements. 

Given the emphasis on understanding reception in a single country context and the 
focus on policy, media content analysis is a useful tool. While newspapers are not the 
only textual sources that could have been used, they are excellent sources for 
understanding public discourse. This is particularly true in the U.S., where journalists 
may play a stronger role in interpreting policy than in Europe, for instance, where 
political parties can have more influence on the discourse (Hallin and Mancini, in Ferree, 
et. al., 2002, p. 81). 

Developing a coding scheme 

A prior literature review of research on PISA reception informed the research questions 
and the construction of the coding scheme. Neuendorf (2002) argues that in order to 
minimize research bias, categories must be developed fully before the coding of data 
commences. However she suggests performing a literature review and a preliminary 
reading of a sample of texts to capture important variables before the 
codebook/codesheets are finalized. A similar process was used in the development of 
the codesheet for the analysis of PISA references in the American media. After reading 
the relevant scholarly literature (both theoretical and empirical), a list of variables for 
coding was developed. Then I scanned a small number of media articles to see if the 
initial list resonated, and if there were other important themes that had been 
inadvertently excluded and added them to the coding scheme. 

It is important to note that the coding categories discussed herein are by no means 
exhaustive, either relative to this specific project or with regard to media content 
analysis in general. All aspects of the research design, from theory to conceptualization 
to operationalization and sampling, are specific to the research questions and the 
researcher's interests. The brief overview here is meant to merely be suggestive of the 
types of questions and categories one might ask when conducting media content 
analysis in a comparative education setting. 
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Here I will highlight three aspects of the codesheet for the PISA media analysis: general 
descriptive categories, framing categories and standing categories. Each ultimately 
provided voluminous data that was used for both quantitative and qualitative analyses 
(see Pizmony-Levy, 2013 and Olafsdottir, 2007 for prior examples of this type of 
analysis). The coding process linked theory to operationalization, and underscored the 
quantity and quality of information media can provide about the national context of 
education politics and policy. 

The first section of the codesheet gathers descriptive information about each newspaper 
article (Figure 1). Some data are responses to open-ended questions, with the exact 
language of the article recorded, and other data are coded numerically, depending on 
the variable type. After the article was given an identification code, the following data 
were recorded: 


Figure 1: Codebook categories for article description 


News Source 

New York Times 

Wall Street Tournal 

Washington Post 

USA Today 

Publication date (Write in: 

Month / day / year) 


Author's name (Write in) 


Headlines/subheads (Write in verbatim) 


Article type 

News 

Feature 

Editorial 

Op-ed 

Other 

Source type 

Print 

Online 

Section 

Front page 

National 

International 

Business 

Feature 

Other 

Visuals 

No 

Yes 

If yes, write in what kind (e.g., 
photograph, chart, , 

graph) 


This information provides a wealth of important data, before the coder even looks at the 
content of the article. For example, these data show which news outlets are publishing 
articles about PISA, allowing for an analysis of whether the media outlet's political 
ideology correlates with PISA coverage. The language of article headlines and sub¬ 
headings can be analyzed for tone and for framing effects. The data on location of the 
article shows whether PISA is addressed simply as a news item around the release of 
results, or incorporated more broadly throughout educational discourse. Indications of 
whether and how media included visual representations of PISA, for instance with 
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graphics of ranking tables, add nuance to how PISA is understood by the media. These 
data are the basis for several findings, to which I return later. 

Next, to analyze the article's content, I developed a two level coding scheme based on 
the concepts of framing and voice to help focus and deepen the analysis. Each article in 
the sample was coded for framing and then re-analyzed for standing and voice 
variables, as I discuss in greater detail below. The two levels of analysis allowed for a 
thorough understanding of how PISA was covered generally and more specifically 
elucidated who was given the authority and legitimacy to speak about PISA and its 
meaning. 

Article-level analysis. Previous literature and pre-coding review had suggested three 
frames that were prominently used to understand PISA. The first was about the overall 
status of education in the United States; the second was about economic status; and the 
third was about the policy environment. I developed thirty-nine variables across the 
three categories to measure the strength of the various frames. Article level variables 
also included the extent to which the references to PISA were rather simple or were 
more detailed; whether or not international assessments other than PISA were 
referenced; and whether or not other country participants on PISA were referenced. 
Figure 2 is a snapshot of the coding scheme for the first frame, the overall status of 
education in the US. For each frame and each sub-category, I coded a dichotomous 
indicator of whether the article included that category. 
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Figure 2: Codebook framing categories for "US education status" 


Frame 

Variable 

US education status 

PISA reflects general education 

status 

US education is in crisis 

Low rank on PISA 

Low rank on other assessments 
Look at China 

Achievement gap 

US 'not competitive' 

US education needs improvement 

Need to improve overall 
economic equity 

improve equity of inputs 
improve equity of outputs 
Need to improve teachers 

improve teacher 

recruitment 

improve teacher training 
improve teacher pav 

Need to improve education 
culture 

General calls for school reform 

PISA does not reflect education status 

PISA data has limits 

Other variables are more important 
US demographics 

US culture 

US politics/political svstem 
Other (write in) 


Speaker-level analysis. In addition to analyzing how educational policy issues were 
framed in relation to PISA, I explored the notion of standing; that is, which actors are 
given a voice in the public debate. In order to understand which actors have voice in the 
discussions about the importance of PISA for U.S. education policy, I coded every actor 
and every speech act in each article. Speech acts included only speech that is quoted 
directly, in quotations, in the articles; paraphrases or restatements of comments from 
people or reports are not included in the analysis. The speakers were categorized into 
eight main groups of affiliations, and then each individual speech act was coded for 
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PISA mentions, tone and framing themes. Figure 3 provides a snapshot of the codebook 
for speaker entries. 


Figure 3: Codebook speaker categories 


Speaker ID 


Speaker name (write in) 


Speaker affiliation 

Government 

US President 

US Secretary of Education 
Federal official 

State level education leader 

OECD 

NGO 

University 

Teacher or teachers 

organization / union 

Parent or parent organization 
Student or student organization 
School leadership 

OTHER (WRITE IN) 


Sampling 

Next, the researcher must address her sampling parameters. As in any research design, 
sampling is critical to answering the research questions of interest. In the United States, 
the most commonly analyzed media sources include the major national newspapers of 
record. In the PISA research example, four of these leading US newspapers, all dailies 
and each considered a paper of record from across the political spectrum, were 
purposefully sampled (see Figure 4). These sources are known for their influence and 
are often used in studies of American discourse (e.g., Fiss and Hirsch, 2005). Sampling 
these sources makes sense, as they are well-known, proffer high editorial standards, and 
are recognized as important news sources for educational elites. However, focusing 
exclusively on these media outlets produces elite bias in the sample. While this may be 
appropriate, as educational elites play an outsized role in policy formation (Domhoff, 
2006) such sampling does not capture the discourse in many other sources (e.g., online 
news feeds, educational blogs, specialized trade publications, smaller regional 
newspapers). The researcher needs to be aware of the ways in which her sampling 
procedures affect her research findings and think carefully about which media sources 
are most appropriate. 
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Figure 4: Overview of media sources 


Source 

Frequency of 

publication 

Political 

orientation 1 

Circulation 2 
(includes both 

print and digital 
circulation) 

New York Times 

Daily 

Left / liberal 

1,865,318 

Wall Street Journal 

Daily 

Right/ conservative 

2, 378,827 

Washington Post 

Daily 

Left of center 

473,462 

USA Today 

Daily 

Centrist 

1,674,306 


'Political orientation of media coverage can be contested. My categorizations are drawn from a 
variety of sources, including Swanson and Barlage (2006), McGann (2013) and Yettick (2009). 
"Circulation figures from Alliance for Audited Media, 2013. 


Using Factiva, a subscription-based search engine owned by the Dow Jones Company, I 
was able to search all four newspapers' databases. I searched for "PISA," "Programme 
for International Student Assessment" and "OECD PISA." Unsure that Factiva was 
actually capturing all articles and blog posts from each media source, I searched the 
online archives of each media source manually, using the same search terms. This 
uncovered several more articles that included the search terms but had not been 
retrieved in the initial searches. The time period for all searches was delimited from 
January 1, 2000 through December 31, 2012. 

After the search was completed and articles downloaded, each article was coded by 
hand by the researcher. There are many software packages that perform content analysis. 
For this analysis, given the size of the sample and the nature of the analysis, human 
coding was an appropriate method. All data were managed in Excel and statistical 
analyses were run using STATA 12.0. 

Analysis of media content 

Media content analysis provides a fascinating window into the construction of PISA as a 
measure of educational excellence in the United States. Looking at public discourse over 
time allows for a systematic yet nuanced understanding of the ways in which PISA is 
being used in educational discourse and educational policy. Some of the most 
interesting findings were unexpected and might have otherwise gone unnoticed, if one 
was simply an interested observer of international assessment results reading the 
newspaper in the United States. 

Take the issue of time. Previous accounts of PISA reception tended to look at 
newspaper coverage of PISA immediately before and immediately after the release of 
results, which happens every three years. Instead, I coded all articles over an entire 
thirteen-year period, 2000-2012, using two temporal variables. By analyzing PISA 
reception over a thirteen year period, the data show that media coverage, and thus PISA 
reception, is not constant. Rather, it shows periods of greater and lesser activity. Using 
the publication dates of the articles in the sample. Figure 5 shows the dramatic increase 
in media coverage by year throughout the first decade of testing. There is a dramatic 
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increase in the sheer number of articles referencing PISA in the American press from 
2010-2012, when fifty-three articles, or 72% of the sample, were published. This was an 
average of more than 17 articles per year, up from 2.3 articles per year through 2009. 
Identifying this pattern was the basis for some of the most informative findings of the 
research. 


Figure 5: Media sample over time 



Source: Green Saraisky, 2015. 


Another temporal variable was the "discourse moment." Following Ferree et. al. (2002), I 
distinguish between three time periods in which the articles are published and coded 
accordingly: articles that were published within a one-month period surrounding the 
release of new PISA results; articles that discuss PISA scores but were published more 
than one month before or after the release date; and articles that refer to PISA but were 
actually about a different educational or social issue. Figure 5 shows the shift in news 
moments over time, suggesting that while PISA results were initially covered in the 
news as news items around the triennial release of results, as the idea of PISA became a 
taken-for-granted measure of educational excellence in the public consciousness, it was 
referenced more consistently throughout the years. 
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Figure 6: Coverage of PISA by discourse moment and by source 



Discourse moment 

■ PISA, within one month PISA, after one month ■ Other ed issues, citing PISA 

The theoretical literature suggests that elites play a key role in the policy process 
(Kingdon, 2011), and the analysis of speech acts bore this out. At the speaker level, the 
discourse is guided by a mere handful of elites: the data show that six speakers provide 
almost 30 percent of the commentary on PISA in the US. There is virtually no public 
voice in the discourse, despite the fact that education is one of the most public of issues - 
virtually all citizens have been in an education system at some point. Teachers, parents 
and students are almost non-existent in the discussion. Instead, a small, highly elite 
group of policy analysts and researchers drive the discourse. Out of the speech acts 
coded, the single most frequently quoted actor in the American media is the European 
head of the Organization for Economic Cooperation and Development, which 
administers PISA 

Limitations and Conclusion 

Media content analysis is not without its limitations. First, there are theoretical 
disagreements about how much influence media have in the policy process, and the 
assumptions of the media's importance in the classic policy studies literature have 
surely been changed with the advent of electronic and social media. Beyond that, 
however, the media can never give a complete picture of why certain policy ideas come 
to the fore, or why certain policy choices are made or not. The public discourse on any 
given education topic represents only one of the many venues in which policy 
deliberation occurs. 

Additionally, media content is produced within an environment of particular 
constraints. Media tend to publish shorter, dramatic stories rather than those that are 
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longer or more nuanced and complex. The pressures of deadlines, competition from 
other news outlets, the limits of reporters' and editors' own knowledge about a given 
topic, and the political leanings of news sources may conspire, together or in 
combination, to limit how education news is covered (Henig, 2008). It is also important 
to distinguish between elite and popular press, and the readership of each. Such 
characteristics place limits on the generalizability of findings, as can research that 
analyzes a limited number of media sources or coverage at only one point in time. 

Nonetheless, if done well, media analysis provides a reliable, valid and replicable 
method for understanding the public context of education. The categorization and 
analysis of media can elucidate trends, beliefs or the Zeitgeist of the times. Media 
content analysis helps to show how education fits into the contemporary polity as a 
whole. Media discourse plays a prominent role in issue identification and issue framing, 
and acts as an important conduit in transmitting information and opinion. While media 
discourse does not represent complete variation in cultural attitudes, beliefs, and values, 
it provides a strong measure of the overarching cultural and political arguments being 
made within a given country, as well as important variations within and across nations. 
While electronic media, including blogs and social media, continue to rapidly change the 
media's role, traditional newspapers remain appropriate sources to provide insight into 
national cultural and political landscapes. 

The empirical examples presented about the reception of PISA in the US media between 
2000 and 2012 exemplify the way media content analysis can be used to produce 
nuanced insights about the construction of educational discourse that might otherwise 
remain latent. While hardly a comprehensive overview, the examples are meant to 
provide some instances of the kind of data and analysis that can emerge from media 
content analysis. 

A recent focus of the comparative education literature has been the way in which ideas 
and global trends have influenced national education policy on the ground. One can 
view the public representation of ideas about education as the outcome of a contest 
about meaning of education. Media are one set of organizations that can show 
researchers what education "means" and how it is interpreted or problematized in a 
particular time and place. Toward that end, media content analysis is a method that 
comparativists can leverage to analyze the ways in which ideas about education shape 
the direction of education policy and discourse. 

If you have any questions or comments on this piece, you may contact Nancy Green Saraisky at 
nlg2004@columbia.edu 
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